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Mixing or combining different elements for getting enhanced version, is 
practiced across various areas in real life. Pan-sharpening is a similar 
technique used in the digital world; a process to combine two images into a 
fused image that comprises more detailed information. Images referred 
herein are Panchromatic (PAN) and Multispectral (MS) images. This paper 
presents a pansharpening algorithm which integrates multispectral and 
panchromatic images to generate an improved multispectral image. This 
technique merges the Discrete wavelet transform (WT) and Intensity-Hue- 
Saturation (IHS) through separate fusing criterion for choosing an 
approximate and detail sub-images. Whereas the maximal local extrema are 
used for merging detail sub-images and finally merged high-resolution image 
is reconstructed through inverse transform of wavelet and IHS. The proposed 
fusion approach enhances the superiority of the resultant fused image is 
demonstrated by quality measures like CORR, RMSE, PFE, SSIM, SNR and 


PSNR with the help of satellite Worldview-II images. The proposed 
algorithm is correlated with the other fusion techniques through empirical 
outcomes proves the superiority of the final merged image in terms of 
resolutions than the others. 
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1. INTRODUCTION 

Most of remote sensing satellites are producing images with high spectral and low spatial or low 
spectral and high spatial resolutions i.e. Multispectral and Panchromatic images respectively. Sharp spectral 
resolution highlights different objects present in the image while main feature of accurate spatial resolution is 
to describe the outline, contents and structure of the image. Neither a multispectral nor a panchromatic image 
is sufficient to best describe the earth images essential for many of the remote sensing applications. High 
spatial and spectral images are suitable for observing the earth images as it involves information like feature, 
shape, structure and different objects. Direct production of such images is inconvenient due to lack of 
bandwidth and onboard storage available at the satellite [1]. Hence, it is necessary to adopt the process of 
combining a high spatial and spectral resolution into one image is also called as Pansharpening [2]. This 
process generates a high-resolution MS image which greatly describes earth images. Important factor to be 
considered is fused image should contain more interpretation capability, also addition of spatial information 
without distracting spectral information [3]. 

The widely used pansharpening techniques are sorted into two key categories: component 
substitution pansharpening technique (CS) and multiresolution analysis pansharpening technique (MRA). 
W. Zang et.al. Investigates a framework for development of image fusion techniques and is used to develop, 
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compare and analysis various fusion techniques [4]. Comprehensive study of various image fusion algorithm 
along with applications are available in the literature [5]-[11]. The earlier is based upon on the exchange of 
element with PAN image. In this, spectral transformation of the MS image is worked as a component. 
Intensity hue saturation (IHS) [12]-[15], Gram Schmidt (GS) [16] and principal component 
analysis [17]-[20] are extensively used component substitution pansharpening techniques. HIS 
pan-sharpening technique works as following steps: first it converts given low resolution multispectral 
images into IHS model then the extracted intensity I is exchanged by high-resolution panchromatic image. 
And finally, inverse transform helps to generate high-resolution multispectral image. But the merged image 
generated using this technique always deficit due to spectral distortion even if contains high spatial 
resolution [21]. 

The latter approach is based on the partition of the images and then reconstruction of new image 
with spatial and spectral details. The several categories of MRA pan-sharpening techniques are decimated 
wavelet transform (DWT) [22], undecimated wavelet transform (UDWT) [23], “à trous” wavelet transform 
(ATWT) [24] and Laplacian pyramid (LP) [25]. In Wavelet transform (WT) pan-sharpening technique, WT 
is applied on both images and decomposed into two sub-images: approximation image which includes 
generalization of the source images and detail image comprises of detail description of the source images. 
Then, the approximate and the detail sub-images are combined to generate a merged image. The resultant 
image obtained through inverse WT. This category of pansharpening technique preserves spectral 
information but with some spatial distortions like blurring [26]. Hybrid approach for image fusion involves 
integration of CS and MRA are investigated in [27]-[31]. In this paper, the proposed hybrid algorithm is 
described as, initially, IHS transform is applied on intensity component of MS image. 

SFIM and wavelet transform is applied on a result of the above step and PAN image to generate 
outcome with multiscale representation. Spectral property of the resultant image is preserved using SFIM. 
Information accommodated in approximate image is distinct from detail sub-image so those need to be 
processed separately. Finally, the inverse wavelet transforms (IWT) as well as an inverse intensity hue 
saturation (IHS) transforms are implemented to get the resultant fusion image. The superiority of the hybrid 
algorithm is proved through the empirical outcomes obtained with the help of Worldview-II images. This 
paper is organized into different modules such as Module II defines the traditional image fusion algorithms. 
Module III represents the proposed hybrid pan-sharpening algorithm. The experimental results with analysis 
for images of Worldview-II dataset are showcased in module IV. At last, conclusion is stated in module V. 


2. RESEARCH METHOD 

This section provides the concepts related to the proposed technique. The proposed algorithm 
consists of a hybrid approach using image fusion techniques like IHS [12]-[15], wavelet transform (WT) 
[22]-[25], smoothing filter-based intensity modulation (SFIM) [32], and local extrema [33]. 


2.1. Image fusion using IHS 
IHS is one of the component substitution technique, where given an image, is transformed into IHS 
from RGB and I component is recovered by given PAN image and at the end resultant image is gained 
through inverse IHS transform as described by the given algorithm below. 
a. Up sample the MS image to PAN image 
b. Transform MS image to IHS model from RGB model 
c. The intensity is considered as, 
I = $Z aM, where Z represents band count, ag = 1/3 for RGB image and M,,denotes k'" band of 
MS image. 
d. The merged high-resolution image can be obtained through,F,, = Mx + (P — I) where Fx is the kth 
band in the fused image. 


2.2. Image fusion using SFIM 


A proportion of a MS and PAN image and it’s a smoothing filtered image used in SFIM helps to 
improve spatial resolution for spectral bands. Fẹ = oe where (r, c) denotes image pixel co-ordinate, 
Mk represents each band of MS image, P signifies PAN images, P mean represents smoothed PAN image and F 
denotes the merged image. The resultant fused image generated using SFIM improves spatial resolution 


whereas retains spectral information. 
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2.3. Image fusion using wavelet Transform (WT) 

Wavelet transform is more attractive technique of image fusion due to its multiscale characteristics 
which are best suited to accomplish different image resolutions. These are classified into discrete and 
continuous categories [13]-[17], [22]. Here it considers a discrete approach. 

1. Decompose MS and PAN image using DWT 


wi = DWT [ MS (r, c)] = [MSy, MSin MSh» MSpn] 
wł = DWT [PAN (r, c)] = [PANy, PAN} , PANn, PANnn] 


where W! (r, c) and W! (r, c) are the wavelet coefficients of MS (r, c) and PAN (r, c) at level 1. 

2. Calculate fused wavelet coefficients Wẹ (r,c) = [Fi Fin, Fp Fan] at scale 1 with conventional 
substitution method [20]. 

3. Reconstruct fused image F! (r, c) at scale 1 using inverse DWT, F!(r, c) = IDWT[W{ (r, c)] 


3. THE PROPOSED METHOD 
This research discusses about the hybrid image fusion algorithm using integration of CS and MRA 
algorithms as CS pansharpening technique encompasses the ability to improve spatial resolution while MRA 
pansharpening technique incorporates the ability to extract adequate spatial details through preserving 
spectral details. The proposed algorithm uses separate fusion rule criterion for approximate and detail sub- 
image. The detail sub-image comprises of the exact description about the image, therefore, the maximum 
local extrema are selected to merge detail sub-images obtained from wavelet transform. The proposed 
algorithm is described to: 
a. Up sample the MS image to PAN image 
b. Transform MS image to IHS model from RGB mode 
c. The intensity I is defined as, I = 1%, «,M, 
where Z represents band count, a% = 1/3 for RGB image and Mẹ denotes k band of MS image. 
LP 
Pmean 
where I represent intensity value, P denotes the PAN image, P mean represents smoothed PAN image, 
and Inewrepresents the calculated new intensity value. 
e. Decompose Inew and PAN image using DWT 


d. Calculate new intensity value as,lnew = 


wi = DWT [new (r, c)] = Unewy news, , Inewpp Inewnn! 
W1 = DWT [PAN (r,c)] = [PANy, PAN), , PAN}, PANpa] 


where Wł (r, c) and W4 (r, c) are the wavelet coefficients of Inew (r, c) and PAN (r, c) at level 1. 
f. Calculate approximate fused wavelet coefficients Wy (m, n) = [Fy] at scale 1 as, 


newy Thewy Cr, c) > PAN} C, c) 
Fy (r, c) = 40.5 * Chewy + PANn), Thewy (r, c) = PAN; (r, c) 
PAN), Thewy ®© c) < PAN) (r, c) 


g. Calculate detail fused wavelet coefficients [Fip , Fh Fp] at scale | as 


1. Selection of detail coefficients are based on contrast between detail and approximation layer which 
I 
is given by Slh; „„ = =h 
g X lew Terre 
2. Similarly compute above values for remaining detail layer coefficients 


i.e. Shl ew? Shhy, ew? Slhpan, Shlpan and Shhpan. 
3. Fused detail coefficients can be obtained by: 


lewy’ Slhy pew ©) > Slhpan(r, c) 
Fm (r, c) = 4 0.5 * (Inewp + PANin), Slhy hew © ©) = Slhpan(r, c) 
PANip, Slhy ew © c) < Slhpan (r, c) 


4. Similarly apply the above fusion rule to get remaining detail layer coefficients i.e. Fpı(r, c) and 
Fhan (r, €) 
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h. Reconstruct fused image F1(r, c) at scale 1 using inverse DWT, F+ (r, c) = IDWT[Fy, Fin , Fa Fon] 
i. The resultant merged image can be reconstructed through inverse IHS transform as, 
Fk = Mx + (F1 (r, c) — I), where Fy is the k band in the fused image. 


4. RESULTS AND ANALYSIS 

In this paper, dataset generated by a high-resolution WorldView-II satellite is used to verify the 
effectivity of the hybrid algorithm. This satellite provides multispectral and panchromatic images with 
1.84-m and 0.46-m resolution correspondingly. The set of images used are available on the website 
http://www.datatang.com/data/43234. This dataset is created using Digital Globe and organized by Beijing 
key laboratory of digital media, Beihang University [21]. The images of this dataset describe the area of city, 
coastal, bridge, etc. 


4.1. Evaluation indicators 

This paper uses quality metrics like correlation coefficient, signal to noise ratio, peak signal to noise 
ratio, structural similarity, the root means squared error and percentage fit error for comparison of original 
and resultant images [34]-[38]. Table 1 shows in definition of the variable used in the quality metrics. 


Table 1. Definition of the Variable Used in the Quality Metrics 


Var F X M N p,q u Fia È Orx 
Def Fused Resampled MS No. of No. of Pixel Mean Maximum Standard Covariance 
image image TOWS columns coordinates gray value deviation 


a. The correlation coefficient (CC) [37], [38] can be used to measure a presence of spectral information. 
The resultant value of equation reveals the correlativity of fused and MS image and CC is defined as: 


Dp- Vg=alF(p, q) — url, q) — px | 


CC(X, F) = 


pa ye g=ilF(p, q) — urp]? x pai Digs XP, q) — Bx]? 


b. The definition of the peak signal to noise ratio (PSNR) [42], [43] is represented a: 


F2 
PSNR(X, F) = 10lg E 
MKN Lp=1 Lq=ilF(p, a) — H(p, q)]? 


Higher value of PSNR indicates noise in the merged image. 
c. The structural similarity (SSIM) [39], [42], [43] is defined as: 


— _ @pepxtc1)(20px+c2) 
SSIM(X, F) = (up? +u%+c1)(op2+0%+c2) 


SSIM is used to find similarity among MS and fused image; More the SSIM value then the chance to get 
similar images is also more. 

d. The root mean squared error (RMSE) [37], [38] measures difference between two images F and MS. The 
smaller difference specifies improved fusion result. It is represented as: 


RMSE(X, F) = — 


MxN 


p=1 Lg=11F(p, q) — Hx(p, a)]? 


e. Percentage fit error (PFE) [38] calculates the proportion of norm of the gap among the MS image and 
merged image to the norm of the MS image. The less value of PFE specifies improved fusion result. It is 
computed as: 


— norm(x(p.q)-F(p.a)) 
PFE(X, F) = — omeo) * 100 
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f. Signal to noise ratio (SNR) [38] is sensitivity measure of the image. The higher value is most preferable 
as it indicates similarity of MS and fused image. 


M_yN 2 
SNR(X F) = 101840 ( IM DN- [F(p,a)] ) 


EDL DN [F(p,q)-ux(p.q)]? 


4.2. Experimental results 

This paper also considers execution of other image fusion techniques like IHS [12]-[15], 
DWT [24]-[26], [39]-[42], BT [43] and quality measures like RMSE, PFE, MAE, CORR, SNR, PSNR, QI 
and SSIM for two different groups of images where each group contains 2 pairs of input images. This 
experiment and implementation is conducted in MATLAB2011a on a Windows8 computer. 

Experimental results shown in Tables 2-5 supports that the proposed method provides an improved 
result than the other techniques like IHS, BT, and DWT. The highlighted experimental results indicate the 
superiority of the technique in an evaluation. It also indicates that the hybrid method improves the result with 
RMSE, PFE, MAE, SNR, PSNR, and SSIM for both sets of images but not able to improve the result for 
some images in terms of CORR present in Table 4. 

We also observed that our proposed method generates a resultant image with high spectral and 
spatial resolution from human eye’s visual perception as compared to others. The value of RMSE, PFE, SNR, 
PSNR, and SSIM specified in Tables 2-5 verifies that the proposed algorithm works best for the given set of 
images over the other pansharpening algorithms. It is observed that DWT slight edge over the proposed 
algorithm for Worldview-II seaside images of set 1 on correlation parameters, shown in Table 4. 


Table 2. Quality Measure Analysis with the Worldview-II Urban Images Set 1 
Data Shown in Figure 1 


Measures Methods 

IHS BT DWT Proposed 
RMSE 0.1431 0.2445 0.1286 0.1021 
PFE 37.582 64.2087 33.7824 26.8095 
MAE 0.1072 0.1898 0.0905 0.0748 
CORR 0.9399 0.6417 0.953 0.9671 
SNR 8.5003 3.8481 9.4261 11.4342 
PSNR 56.6074 54.2812 57.0703 58.0743 
SSIM 0.9981 0.9917 0.9981 0.9993 


Table 3. Quality Measure Analysis with the Worldview-II Urban Images Set 2 
Data Shown in Figure 2 


Measures Methods 

IHS BT DWT Proposed 
RMSE 0.13 0.3206 0.1108 0.1024 
PFE 26.3652 65.0246 22.4757 20.7641 
MAE 0.0955 0.2687 0.077 0.072 
CORR 0.9683 0.6269 0.9776 0.9796 
SNR 11.5793 3.7384 12.9657 13.6537 
PSNR 57.0244 53.1039 57.7176 58.0616 
SSIM 0.9988 0.9857 0.9988 0.9995 


Table 4. Quality Measure Analysis with the Worldview-II Seaside Images Set 1 
Data Shown in Figure 3 


Measures Mighods 

IHS BT DWT Proposed 
RMSE 0.1246 0.299 0.0914 0.0909 
PFE 28.243 67.7631 20.7233 20.6157 
MAE 0.1003 0.2416 0.0578 0.0665 
CORR 0.9611 0.5889 0.9806 0.979 
SNR 10.9817 3.3801 13.6708 13.716 
PSNR 57.2076 53.4068 58.5521 58.5747 
SSIM 0.9985 0.9875 0.9991 0.9995 
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Table 5. Quality Measure Analysis with the Worldview-II Seaside Images Set 2 


Figures 


1-4 (i) 


Data Shown in Figure 4 


Measures Methods 
IHS BT DWT Proposed 
RMSE 0.1007 0.2141 0.0824 0.0622 
PFE 30.4913 64.8302 24.9711 18.8593 
MAE 0.0873 0.176 0.0576 0.0487 
CORR 0.9586 0.631 0.9734 0.9832 
SNR 10.3164 3.7644 12.0512 14.4894 
PSNR 58.134 54.8579 59.0013 60.2205 
SSIM 0.9987 0.9934 0.9991 0.9996 
indicate the resampled low-resolution MS 


images 


of Worldview-II. 


The corresponding high spatial resolution PAN images are displayed in Figures 1-4 (ii). The resultant fused 
image obtained by traditional fusion methods like BT, IHS, DWT and the proposed method are presented in 
Figures 1-4 (iii), (iv), (v) and (vi). respectively. Resultant fused image after applying proposed algorithm 
improves the clarity of the image with more accurate geographical details are observed in Figures 1-4 (vi). 


(iii) 


Figure 1. Pan-sharpening outcomes about Worldview-II urban images set 1: (i) Input MS image after 
resampling (ii) Input PAN image (iii) BT (iv) IHS (v) DWT (vi) Proposed hybrid technique 
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Figure 2. Pan-sharpening outcomes about Worldview-II urban images set 2: (i) Input MS image after 
resampling (ii) Input PAN image (iii) BT (iv) IHS (v) DWT (vi) Proposed hybrid technique 
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Figure 3. Pan-sharpening outcomes about Worldview-II seaside images set 1: (1) Input MS image after 
resampling (ii) Input PAN image (iii) BT (iv) IHS (v) DWT (vi) Proposed hybrid technique 
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Figure 4. Pan-sharpening outcomes about Worldview-II seaside images set 2: (1) Input MS image after 
resampling (ii) Input PAN image (iii) BT (iv) IHS (v) DWT (vi) Proposed hybrid technique 


5. CONCLUSION 

The research work is focused on implementation of a hybrid image fusion algorithm which merges 
CS approach and MRA approach to overcome drawbacks of individual approach. This hybrid approach is 
designed to enhance superiority of images. Separate fusion rules are adopted in this hybrid approach for the 
approximate and the detail sub-images. The research work also involves comparison of hybrid pan- 
sharpening technique with some traditional algorithm like IHS, DWT, BT. Experiments are conducted on 4 
sets of images from Worldview-II. The Superiority of a resultant merged image is calculated in terms of 
RMSE, PFE, MAE, CORR, SNR, PSNR, and SSIM as shown in Table 2-5. Empirical outcomes demonstrate 
that the hybrid algorithm gives improved fusion result than the others. The fused image obtained from the 
hybrid algorithm progresses best in resolution, so it can be used further for image processing applications for 
classification, feature extraction etc. 

Current work can be altered in future through implementing different fusion rules for merging 
approximate and detail sub-images. Also, this work can be extended by developing a different hybrid 
approach for satellite images generated from different satellites. 
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